Scaling Up a Distributed Computing Of Similarity Coefficient with Mapreduce

The work presented in this paper addresses the design and implementation of a Hadoop application and the experiments performed with this application in order to compute the Jaccard similarity metrics for two very large graphs. The algorithm involved uses the MapReduce programming model, whose aim is to distribute the computing process over several machines in order to reduce the overall running time. As a distributed programming model, MapReduce is one of the most important techniques behind Cloud computing metaphor, focused on data intensive computing in clustered environments. Hadoop open source framework provides to developers a Java API for implementing applications based on MapReduce programming paradigm. In this philosophy, the main task is divided into several smaller subtasks that can be executed or re-executed on any node in the cluster. The experimental results presented in this paper were obtained after performing various tests over two large data sets (WEBSPAM-UK 2007 and Slashdot) on a distributed cluster.